Skip to content

fix(evaluation): avoid crash on first turn with empty user content#6405

Closed
anxkhn wants to merge 1 commit into
google:mainfrom
anxkhn:fix/eval-per-turn-empty-user-content
Closed

fix(evaluation): avoid crash on first turn with empty user content#6405
anxkhn wants to merge 1 commit into
google:mainfrom
anxkhn:fix/eval-per-turn-empty-user-content

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

No existing issue. Describing the bug below per the issue-template structure, as
CONTRIBUTING permits for a PR without an associated issue.

2. Or, if no issue exists, describe the change:

Problem:

PerTurnUserSimulatorQualityV1._evaluate_first_turn crashes with
AttributeError: 'NoneType' object has no attribute 'strip' when the first
invocation's user_content is a non-None Content whose parts are empty
(parts=[]) or None (types.Content()).

The method guards only the first_invocation.user_content is None case, then
does:

score = int(
    get_text_from_content(first_invocation.user_content).strip()
    == conversation_scenario.starting_prompt.strip()
)

get_text_from_content (llm_as_judge_utils.py) returns None when the content
has no text parts, because if content and content.parts: falls through for
empty/None parts:

if content and content.parts:
    return "\n".join([p.text for p in content.parts if p.text])
# implicit return None otherwise

So .strip() is called on None and raises. This is reachable through the
public Evaluator.evaluate_invocations, which calls
_evaluate_first_turn(actual_invocations[0], ...): any evaluated conversation
whose first user turn was built with empty parts (or carries only non-text parts
after text extraction returns None) crashes the evaluation instead of producing
a result.

Reproduction (text extraction results that reach _evaluate_first_turn):

                    parts=[]  ->  None      # old code: .strip() -> AttributeError
             types.Content()  ->  None      # old code: .strip() -> AttributeError
                   text part  ->  'hi'       # scored normally
     function_call-only part  ->  ''         # scored (legitimately FAILED)

Solution:

Extract the text once and return a NOT_EVALUATED PerInvocationResult when it
is None, before the comparison. This mirrors the two guards already in this
file: the user_content is None guard immediately above, and
_convert_llm_response_to_score's if response_text is None or not response_text: return AutoRaterScore().

user_text = get_text_from_content(first_invocation.user_content)
if user_text is None:
    return PerInvocationResult(
        actual_invocation=first_invocation,
        eval_status=EvalStatus.NOT_EVALUATED,
    )

score = int(
    user_text.strip() == conversation_scenario.starting_prompt.strip()
)

The fix is behavior-preserving for every case that did not crash: the guard is
is None only, so a non-text part (text extraction returns "", not None) is
still scored and legitimately marked FAILED. Only the two previously-crashing
inputs (parts=[], types.Content()) now return NOT_EVALUATED.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added a parametrized regression test
test_evaluate_first_turn_not_evaluated_when_user_content_has_no_text covering
types.Content(role="user", parts=[]) and types.Content(), asserting the
result is NOT_EVALUATED (score None) instead of raising.

$ pytest tests/unittests/evaluation/simulation/test_per_turn_user_simulation_quality_v1.py -q
33 passed

$ pytest tests/unittests/evaluation -q
486 passed

Effectiveness check: temporarily reverting the new guard makes the two new
parametrized cases fail with the exact
AttributeError: 'NoneType' object has no attribute 'strip'; restoring the guard
makes them pass. The test is not vacuous.

Manual End-to-End (E2E) Tests:

Not applicable; this is a unit-level crash guard in the evaluation module,
covered by the added regression test. The behavior is demonstrated by the
reproduction table above (empty/None parts extract to None, which previously
hit .strip()).

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas. (No new
    comments needed; the guard mirrors the existing ones directly above/below it.)
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end. (Unit-level fix; covered by
    the regression test rather than an E2E flow.)
  • Any dependent changes have been merged and published in downstream modules. (None.)

Additional context

The two new NOT_EVALUATED cases are consistent with how the aggregator already
handles non-evaluated turns (num_evaluated == 0 yields an EvaluationResult
without an overall_score), so a conversation whose only turn is empty no longer
crashes and simply contributes nothing to the score.


Diff summary

  • src/google/adk/evaluation/simulation/per_turn_user_simulator_quality_v1.py
    (+10/-2): compute user_text = get_text_from_content(...) once; return
    PerInvocationResult(eval_status=NOT_EVALUATED) when user_text is None; reuse
    user_text in the comparison.
  • tests/unittests/evaluation/simulation/test_per_turn_user_simulation_quality_v1.py
    (+26): parametrized regression test over types.Content(role="user", parts=[])
    and types.Content().

`_evaluate_first_turn` guarded only against a `None` `user_content`, then
called `get_text_from_content(...).strip()`. When `user_content` is a
non-None `Content` whose parts are empty or `None`, `get_text_from_content`
returns `None`, so `.strip()` raised `AttributeError: 'NoneType' object has
no attribute 'strip'`.

Compute the extracted text once and return a `NOT_EVALUATED` result when it
is `None`, mirroring the existing guard for a `None` `user_content` above it
and the `None`/empty guard in `_convert_llm_response_to_score`. Add a
regression test covering `parts=[]` and `parts=None`.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Jul 16, 2026
copybara-service Bot pushed a commit that referenced this pull request Jul 16, 2026
@adk-bot

adk-bot commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Thank you @anxkhn for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit c9bacd4.

Closing this PR as the changes are now in the main branch.

@adk-bot adk-bot added the merged [Status] This PR is merged label Jul 16, 2026
@adk-bot adk-bot closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation merged [Status] This PR is merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants